1. Field of the Invention
The present invention relates generally to an improved data processing system. More particularly, the present invention provides a mechanism for aggregating data in a way that permits data to be deleted efficiently, while minimizing the overhead necessary to support bulk deletion of data.
2. Description of the Related Art
Early file systems were designed with the expectation that data would typically be read from disk many times before being deleted. Therefore, on-disk data structures were optimized for reading of data. However, as main memory sizes increased, more read requests could be satisfied from data cached in memory. This motivated file system designs that optimized write performance rather than read performance. However, the performance of such system tends to suffer from overhead due to the need to garbage collect current, i.e. “live,” data while making room for areas where new data can be written.
New types of systems are evolving in which, in addition to reading and writing of data, creation and deletion of data are important factors in the performance of the system. These systems tend to be systems in which data is quickly created, used and discarded. These systems also tend to be systems in which the available storage system resources are generally fully utilized. In such systems, the creation of data and deletion of this data is an important factor in the overall performance of the system.
However, known file systems, which are optimized for data reads or, alternatively, data writes, do not provide an adequate performance optimization for this new breed of systems. Previous file systems teach a method whereby a sequence of objects is stored in a set of storage segments. See “Position: Short Object Lifetimes Require a Delete-Optimized Storage System,” by Douglis et al., 11th ACM SIGOPS European Workshop, September 2004, which is hereby incorporated by reference. Typically, such segments are fixed in size and pre-allocated. At any given time, a plurality of segments are available for storing newly written objects, with each segment holding objects with similar retention attributes, specifically a retention value and retention decay function. When an object is to be stored, a then-in-use segment is the first target of the store operation. Although the segment may be empty, typically, the segment already holds a plurality of other objects. Therefore, it would be advantageous to have a system and method for a mechanism that aggregates data in a way that permits data to be deleted efficiently, while minimizing the overhead necessary to support bulk deletion of data.